Lightning to use sync v2 formats by taylordowns2000 · Pull Request #4722 · OpenFn/lightning

taylordowns2000 · 2026-05-08T07:49:39Z

Description

Closes #4718 .
Closes #4727 .

Switches Lightning to the v2 (portability spec) YAML format for the four user-facing surfaces below. The wire shape is now fully aligned with the OpenFn portability spec: @openfn/kit/packages/lexicon/portability.d.ts.

1. Canvas → "View workflow as code"

CodeViewPanel serializes the live workflow state to v2 YAML via assets/js/yaml/format.ts (delegates to assets/js/yaml/v2.ts).

1a. Export button

Same code path — the Download button writes the v2 YAML the panel renders.

1b. Publish as template button

TemplatePublishPanel writes new templates in v2. Existing v1 templates remain readable on the Pick Template path (see 2a).

2. Create new → "Import code as workflow"

YAMLImportPanel uses the format-aware façade: detects v1 vs v2 from pasted/uploaded YAML and dispatches to the matching parser. Empty-editor placeholder shows the v2 shape so new users land on the new format.

2a. Pick template

TemplatePanel runs every template through the same façade. v1 templates load (lenient v1 parser), v2 templates parse strictly. A test asserts that the v1 and v2 versions of the same scenario produce structurally equivalent workflow state.

3. Settings → "Export project" button

Projects.export_project/2 calls Lightning.Workflows.YamlFormat.serialize_project/2, which emits v2 YAML. The old v1 emitter (Lightning.ExportUtils) is removed.

4. GitHub sync — block sandbox/ancestor branch collisions

A sandbox project can no longer link to the same (repo, branch) as any of its ancestors:

Projects.ancestor_ids/1 walks the parent_id chain via a recursive CTE.
ProjectRepoConnection.changeset/2 validates against ancestor branches and surfaces an inline error in the Save form.
VersionControl.create_github_connection/2 re-checks the same condition transactionally so a concurrent insert can't race past the validation.

Spec coverage

Fully implemented from the spec:

Type	Field	Notes
`ProjectSpec`	`id` (required)	hyphenated project name
`ProjectSpec`	`name?`, `description?`
`ProjectSpec`	`workflows: WorkflowSpec[]`	YAML sequence
`ProjectSpec`	`credentials?: Credential[]`	YAML sequence; entries without resolvable owner email are dropped
`ProjectSpec`	`collections?: string[]`	YAML sequence of names, alphabetically sorted
`WorkflowSpec`	`id?`, `name?`	`id` is the hyphenated name
`WorkflowSpec`	`steps: Array<Job \| Trigger>`	unified array
`Step`	`id?`, `name?`, `next?`
`StepEdge`	`boolean \| string \| ConditionalStepEdge`	parser accepts all three; emitter prefers the object form
`ConditionalStepEdge`	`condition?` (JS body), `label?`, `disabled?`
`Trigger`	`enabled?`
`Credential`	`name`, `owner` (both required)
`Job`	`adaptors?: string[]`	Lightning state has a single adaptor; collapsed/wrapped to a 1-element array
`Job`	`expression?`
`Job`	`configuration?: object \| string`	string form only (Lightning credentials are `<email>\|<name>` strings)

Edge condition mapping

Lightning's internal condition_type enum ↔ spec condition (a JS expression body):

Lightning enum	Wire `condition`
`:always`	omitted (or `next: <target>` shortcut)
`:on_job_success`	`'!state.errors'`
`:on_job_failure`	`'!!state.errors'`
`:js_expression`	`<user JS body>` (literal block when multi-line)

Parser also accepts the spec's shortcuts:

next: { foo: true } → :always
next: { foo: false } → :js_expression body "false" (Lightning has no :never)
next: { foo: "<js>" } → :js_expression body verbatim

Lightning extensions (documented)

The portability spec marks Trigger fields explicitly TODO (// TODO a trigger supports many more keys which the spec must support). Until the spec author fills these in, we keep two trigger-only extensions:

type: discriminator (webhook / cron / kafka)
openfn: block carrying cron expression, cron cursor reference, webhook reply mode, and Kafka config

These are commented as Lightning extensions in both v2.ex and v2.ts.

What does NOT work yet (out of scope for this PR)

Server-side YAML parsing. This PR only emits v2 on the server. The provisioning controller still accepts the legacy provisioner-shape JSON unchanged. Accepting v2 YAML/JSON over the API (and the matching @openfn/cli integration) is intentionally deferred — there is no Phase 5 import bridge in this branch.
Spec fields not yet mapped: WorkflowSpec.globals, WorkflowSpec.start, WorkflowSpec.options, Step.previous, Job.configuration as object. Lightning has no internal equivalents today.
test/integration/cli_deploy_test.exs is broken by the export cutover (it compares against a v1 fixture). Tagged :integration and disabled by default; will be updated when the @openfn/cli suite is next exercised.

Validation steps

Open a workflow → Code panel → confirm id:, name:, steps: array, no jobs:/edges:, jobs use adaptors: [...], expression bodies under expression: |.
Click Download → file matches the panel content.
Publish as template → the saved template's code is v2.
Pick Template → both a v1 (legacy) and a v2 template load into a working canvas.
Import code → paste v1 YAML, paste v2 YAML, both import.
Project settings → Export → downloaded file has top-level id:, workflows: is a sequence (- id: …), credentials: is a sequence of {name, owner}, collections: is a sequence of names.
Edge conditions: a workflow with on-success / on-failure edges round-trips through Code panel → reload as !state.errors / !!state.errors.
GitHub sync: connect a parent project to repo:main, create a sandbox of it, attempt to connect the sandbox to repo:main → inline error, Save button disabled. Switch the sandbox to repo:dev → Save enabled.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

I have used Claude Code
I have used another model
I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

I have performed an AI review of my code (we recommend using /review
with Claude Code)
I have implemented and tested all related authorization policies.
(e.g., :owner, :admin, :editor, :viewer)
I have updated the changelog.
I have ticked a box in "AI usage" in this PR

codecov · 2026-05-09T11:22:21Z

Codecov Report

❌ Patch coverage is 86.78571% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.85%. Comparing base (7cedc62) to head (e383134).

Files with missing lines	Patch %	Lines
lib/lightning/workflows/yaml_format/v2.ex	84.61%	36 Missing ⚠️
...ghtning/version_control/project_repo_connection.ex	96.15%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4722      +/-   ##
==========================================
- Coverage   89.93%   89.85%   -0.09%     
==========================================
  Files         444      444              
  Lines       21979    22110     +131     
==========================================
+ Hits        19767    19866      +99     
- Misses       2212     2244      +32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Broaden the GitHub-sync repo+branch guard from "no ancestor may share" to "no project in the same tree may share" (root, sandboxes, siblings, cousins all share one (repo, branch) namespace), and move the enforcement from a check-then-insert pattern in Elixir to a Postgres unique index. Two concurrent transactions in the same project family can no longer both succeed at READ COMMITTED. - Add Lightning.Projects.root_id/1 walking parent_id to the top. - Migration adds project_repo_connections.root_project_id (NOT NULL after backfill) and a unique index on (root_project_id, repo, branch). - ProjectRepoConnection sets root_project_id on insert, declares unique_constraint(:branch, name: ...), and exposes tree_unique_violation?/1 so the boundary can translate the constraint violation back to :branch_used_in_project_tree. - VersionControl.create_github_connection drops its in-memory pre-flight check and relies on the unique index for race-safety. - Tests cover sibling/cousin collisions, the renamed error, and a direct INSERT race that's only resolvable by the index. Co-authored-by: Claude <noreply@anthropic.com>

github-actions · 2026-05-09T14:53:16Z

Security Review ✅

S0 (project scoping): New root_id/1 walks parent_id via CTE keyed on the caller's project; tree_branch_conflict?/4 filters by root_project_id; validate event in github_sync_component.ex:30 now puts project_id from socket assigns rather than form params — a tightening, not a regression. The V2 YAML serializer is reached only through Projects.export_project/3, whose two callers (downloads_controller.ex:14 and provisioning_controller.ex:191) still gate on Permissions.can/4 before calling.
S1 (authorization): No new web-layer mutating actions; existing can_install_github / can_initiate_github_sync gates on save, delete-connection, reconnect, initiate-sync are unchanged, and the new validate path performs no DB writes.
S2 (audit trail): create_github_connection still wraps the insert and Audit.repo_connection(:created, user) in a single Repo.transact; the new insert_repo_connection/1 only translates the unique-index violation into :branch_used_in_project_tree and does not bypass the audit step.

taylordowns2000 · 2026-05-10T09:14:20Z

@josephjclark , @midigofrank , sorry... dug deeper and this needs a few more tweaks. will be ready for review shortly.

josephjclark · 2026-05-10T10:44:58Z

High level QA notes for this branch:

Export workflow yaml to a file and execute through CLI
Copy a workflow yaml from production (with the old format) and import as a new workflow on this branch. Should be identical
Copy a workflow yaml from this branch (with the new format) and import as a new workflow
Export a project, paste the yaml locally, run a workflow through the CLI

I've run a smoke test and it looks great - the format looks right and I can run a workflow yaml through the latest CLI version

EDIT: QA looks good! Just one issue executing from the project, see below

josephjclark · 2026-05-10T10:51:25Z

@taylordowns2000 I notice an issue here where when viewing workflow yaml, the credential is not included. It's fine when exporting the project.

But this affects main too! So I'm going to raise it out as a bug, No action needed here.

taylordowns2000 · 2026-05-10T11:57:46Z

@taylordowns2000 I notice an issue here where when viewing workflow yaml, the credential is not included. It's fine when exporting the project.

But this affects main too! So I'm going to raise it out as a bug, No action needed here.

This is super interesting @josephjclark - the reason is likely that credentials are not included in templates and "view" is the precursor to "publish as template". How should we handle this?

(a) Leave it as is
(b) Show the credential in view, strip it out in publish?
(c) Somehow leave it in templates, and strip it out on create?
(d) Something else?

josephjclark · 2026-05-11T11:23:03Z

@taylordowns2000 I'll take that convo over to #4731

josephjclark · 2026-05-11T11:30:20Z

@taylordowns2000 one change I'm going to recommend on both sides: let's add something like schema_version to the top of the project.yaml (or, if a workflow.yaml is standalone, let's add it there too).

The CLI currently writes a cli.version: 2 flag to help differentiate state v1 and state v2 files. But after this work that's not just a CLI thing - it's global.

This helps tools like the CLI and maybe later the provisioner understand and anticipate the structure they are about to parse.

The practical upshot of this is that the CLI is erroring if you do openfn checkout project-spec.yaml from a downloaded spec.That's a result of a bad assumption in the CLI code - but without a version flag I'm struggling to override that assumption.

Now, we should probably align that version with the portability spec v4. And we should probably considering semvering it (at least major and minor, maybe not patch)

So I recommend:

Top level schema_version key
Lightning needs to set this on project and workflow export
Set the version value to 4.0
Patch CLI to synchronise to the same version number

This will also make it much easier to coordinate the next schema change we want to make.

BTW I've made a separate issue for the "should portability spec be a JSON schema" thing. Still want to defer that for that though

* Emit schema_version: '4.0' in v2 workflow and project YAML Adds a top-level `schema_version` key after `id` and `name` in both the TypeScript workflow serializer and the Elixir workflow/project serializer. The v2 JSON schema is extended to permit the key so round-trip parsing still passes its own validation. Parsers remain lenient — files without the key continue to parse unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * formatting on yaml output * add schema_version to v1 --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

josephjclark · 2026-05-12T09:36:08Z

@midigofrank This is ready for you 👍

…ization

midigofrank

@josephjclark @taylordowns2000 I'm worried about the provisioner api. This PR replaces the yaml content being downloaded in the endpoint with v2 format, this will break older clis (not sure which versions).

I'll be happy to approve this if there is already a plan in place to handle legacy support

* restore v1 spec emitter * update test * force lowercase ids in v2 * lowercase in frontend

josephjclark · 2026-05-12T16:41:31Z

@midigofrank great catch on that. The provisioner API should not have broken, that should all sit on the v1 stuff, invisible to users.

I've squashed a change which means:

v1 CLI commands (including GH sync) all work great out of the box
The UI exports use the v2 format and are compatible with cli checkout and cli exectute

So we should have working legacy automations, but a clean and tidy default v2 path going forward.

I haven't looked closely at the GH sync stuff so I think I'll do that tomorrow

josephjclark · 2026-05-22T09:34:14Z

@midigofrank has raised a really good flag

The etophia deployment works by:

export the projetct yaml (v1 format)
paste it into GH (I think?)
Run the deploy script, which uses v1 deploy

Obviously when we export the project in v2 style, this process will break.

Are we aware of anyone else who uses the export function like this?

options:

Add support to the v1 deploy to read a v2 spec file (not technically that hard, but I don't want to touch v1 if I can help it). Requires ethiopia to bump CLI version.
Add a small button to export the project in v2 format (or when you hit export, ask users what format they want). I don't love it.
Update ethiopia to use the v2 deploy command. Shouldn't be a big API change. But would need to coordinate the update (script and platform need updating together), and honesly v2 deploy from spec needs more testing

Option 1 feels like the most elegant here. All we've had to do in the CLI is convert formats on the fly - which we have good robust code for.

josephjclark · 2026-05-22T09:36:55Z

Related, see this: OpenFn/kit#1421

josephjclark · 2026-06-22T13:09:34Z

      # Lets use the updated spec
-      specPath = Path.expand("test/fixtures/canonical_update_project.yaml")
+      specPath =
+        Path.expand("test/fixtures/portability/v1/canonical_update_project.yaml")


TODO:

update the CLI to support v2 spec in v1 sync

duplicate this test to use v2 spec

…ization Resolved conflicts: - CHANGELOG.md: merged unreleased entries from both branches - projects.ex: kept both root_id/1 and list_descendants/1 - projects_test.exs: kept describe blocks from both sides - export_utils.ex + webhook_reply fixture: kept deleted (V2 refactor) - yaml/util.ts: kept V2 pass-through refactor Note: channels and webhook_response_config export are not yet ported to the V2 serializer; the corresponding ported tests fail until a follow-up.

Brings two main-branch project-export features into the V2 portability format that the v2-sync refactor had left unported: - channels: emitted as a YAML array of objects (name, destination_url, enabled, destination_credential). The destination_credential reference uses V2's email|name credential-key form, consistent with a job's configuration: field. - webhook_response_config: nested under the webhook trigger step with the success_code / error_code the user has set; omitted when unset. Mirrored on the frontend (assets/js/yaml/v2.ts + schema) for webhook_response_config serialize/parse parity (channels are project-level, backend-only). The inherited tests asserted the v1/main shapes (a triggers: map and hyphenated credential refs); rewritten to the V2 steps-array shape. Also updated two pre-existing project-export tests that hadn't been adjusted when schema_version emission was added.

- Preserve webhook_response_config on import: the webhook branch of convertWorkflowSpecToState dropped the configured success/error status codes on every v2 import, unlike the cron and kafka branches. - Default a trigger's enabled to false on parse, matching the serializer, the Elixir emitter, and the Trigger schema default; parse previously defaulted to true, so the two directions of the module disagreed. - Round-trip JS conditions that collide with named condition literals: a js_expression body such as !state.errors is now preserved instead of being rewritten to on_job_success and dropping the expression. - Add `|` to the quote whitelist so credential keys (email|name) and `||` conditions stay unquoted, matching the Elixir emitter and avoiding byte-different YAML for the same workflow. Correct the stale module-doc comments and add round-trip tests covering each case.

The v2 project export preloaded the full credential chain (project_credential -> credential -> user) on every job and every channel, duplicating the project-level project_credentials that are already loaded. Build a project_credential_id -> "owner|name" lookup once from the project's credentials and resolve each job's configuration: and each channel's destination_credential: by id. workflow_struct_to_canonical now takes a :credential_keys option; serialize_workflow/1 has no project context, so it omits the option and falls back to the job's preloaded project_credential. Drop the redundant nested credential preloads from preload_project_for_export. Add a test covering a job whose configuration resolves from the project credentials by id.

…to v2-sync-finalization

concept for #4718

840677e

github-project-automation Bot added this to Core May 8, 2026

github-project-automation Bot moved this to New Issues in Core May 8, 2026

taylordowns2000 added 7 commits May 8, 2026 10:36

reduce

eed28ab

match spec

0c645a8

dialyzer and credo

c362f6e

update tests

99e94a3

new spec

c316e80

do less

f16baff

fixtures

e7cf054

taylordowns2000 and others added 5 commits May 9, 2026 13:46

tests

116aa14

deeper tests

cb73fa6

tests

2baa674

comment

0855d90

taylordowns2000 marked this pull request as ready for review May 9, 2026 14:50

taylordowns2000 requested review from josephjclark and midigofrank May 9, 2026 18:01

taylordowns2000 added 7 commits May 9, 2026 20:06

cl

682c6ac

review touchups

f89c2c7

trim

31547b0

more tests

cf5cad7

remove stale comment

2a3456c

call it portability, not yaml

d9d1ed6

remove unused

ad694a7

taylordowns2000 marked this pull request as draft May 10, 2026 09:14

v2 deploy test

a34ab28

reduce

5c8b740

simpler test formats

2103dc4

taylordowns2000 marked this pull request as ready for review May 10, 2026 11:12

taylordowns2000 added 2 commits May 10, 2026 14:19

bump cli version

0c200ca

Merge branch 'main' into v2-sync-finalization

e383134

This was referenced May 11, 2026

Github sync with v2 CLI: can only pull one project #4658

Closed

Portability: explicitly serialize to version 4.0 #4738

Merged

josephjclark approved these changes May 12, 2026

View reviewed changes

Merge branch 'main' of github.com:OpenFn/lightning into v2-sync-final…

164c0bf

…ization

midigofrank requested changes May 12, 2026

View reviewed changes

github-project-automation Bot moved this from New Issues to In review in Core May 12, 2026

v2 portability: restore legacy provisioner (#4743)

f5078bd

* restore v1 spec emitter * update test * force lowercase ids in v2 * lowercase in frontend

taylordowns2000 mentioned this pull request Jun 5, 2026

Prevent sandbox from using same Github repo:branch as parent #4727

Open

josephjclark reviewed Jun 22, 2026

View reviewed changes

midigofrank added 5 commits June 26, 2026 17:52

Merge branch 'v2-sync-finalization' of github.com:OpenFn/lightning in…

5de88c6

…to v2-sync-finalization

midigofrank mentioned this pull request Jun 29, 2026

Update v2 sync with main #4903

Closed

7 tasks

Uh oh!

Conversation

taylordowns2000 commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

1. Canvas → "View workflow as code"

1a. Export button

1b. Publish as template button

2. Create new → "Import code as workflow"

2a. Pick template

3. Settings → "Export project" button

4. GitHub sync — block sandbox/ancestor branch collisions

Spec coverage

Edge condition mapping

Lightning extensions (documented)

What does NOT work yet (out of scope for this PR)

Validation steps

AI Usage

Pre-submission checklist

Uh oh!

codecov Bot commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Security Review ✅

Uh oh!

taylordowns2000 commented May 10, 2026

Uh oh!

josephjclark commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josephjclark commented May 10, 2026

Uh oh!

taylordowns2000 commented May 10, 2026

Uh oh!

josephjclark commented May 11, 2026

Uh oh!

josephjclark commented May 11, 2026

Uh oh!

josephjclark commented May 12, 2026

Uh oh!

midigofrank left a comment

Choose a reason for hiding this comment

Uh oh!

josephjclark commented May 12, 2026

Uh oh!

josephjclark commented May 22, 2026

Uh oh!

josephjclark commented May 22, 2026

Uh oh!

josephjclark Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taylordowns2000 commented May 8, 2026 •

edited

Loading

codecov Bot commented May 9, 2026 •

edited

Loading

github-actions Bot commented May 9, 2026 •

edited

Loading

josephjclark commented May 10, 2026 •

edited

Loading